MEMORANDUM September 10, 2019 
TO: Jessica Chevalier, Secondary Literacy Director 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: ACHIEVE3000®, 2018-2019 


The 2018-2019 academic year was the fifth year that the Houston Independent School District 
(HISD) offered the differentiated reading instruction program, Achieve3000°. Based on an 
analytic sample of about 4,900 6" through 12" grade students who completed both a pre-test 
Lexile assessment within the first 30 days of the start of the school year and a post-test Lexile 
assessment in the final 30 days of the school year, this year’s evaluation of Achieve3000 
assessed the relationship between fidelity of use of the tool—i.e., completion of one to two or 
more reading or writing lessons per week (40 to 80 or more activities for the year), with an 
average first try score of 75 percent or greater—and students’ Achieve3000 Lexile growth and 
grade-specific college and career readiness. 


Key findings include: 

e Of the nearly 111,000 students who had access to Achieve3000 in the 2018-2019 academic 
year, only 402 students (about 8% of the analytic sample) achieved high fidelity of use of the 
program for the year. 

e Lexile point growth was greatest for students who both worked toward the completion of 40 
to 80 or more Achieve3000 activities and who had an average first-try score (AFTS) of 75 
percent or greater on those activities. 

e The average Lexile growth for a student completing 40 activities with an AFTS of less than 
75 percent was about 35 points, while the average Lexile growth for a student completing 40 
activities with an AFTS of 75 percent or greater was about 138 points. 

e The odds of being college and career ready at the end of the year decrease by about one 
percent among students with an AFTS less than 75 percent. 

e The odds of being college and career ready at the end of the year increase by about 4 
percent among students with an AFTS of 75 percent or greater. 


Further distribution of this report is at your discretion. Should you have any further questions, 
please contact me at 713-556-6700. 
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Abstract 

Despite its promise to help teachers meet the needs of diverse learners, district usage of Achieve3000® among 
most students with access to it remains far below the optimal level necessary for them to realize literacy 
growth and college and career readiness. Only a tiny minority of students (402 out of about 111,000) seem 
to be engaging Achieve3000 with fidelity, where fidelity is defined as the number of Achieve3000 activities 
completed over the course of the academic year (i.e., 40 to 80 or more), and an average first-try score (AFTS) 
earned on Achieve3000 assessments (75% or greater). This study examined how the minority using 
Achieve3000 with fidelity fared with regard to their Lexile® growth and college and career readiness relative 
to the majority using Achieve3000 without fidelity. Results show that fidelity of use of Achieve3000 is, net of 
student characteristics, associated with greater Lexile growth. For two students with an AFTS of less than 
75 percent, the one completing 40 activities for the year would gain about 14 Lexile points more than the one 
who completed only one activity. Likewise, for two students with an AFTS of 75 percent or greater, the one 
completing 40 activities over the year would gain about 67 more Lexile points than the one who completed 
only one activity. Fidelity of use of Achieve3000 is also related to higher odds of college and career readiness 
at the end of the school year, controlling for readiness at the beginning of the year and other student 
characteristics. For students with an AFTS less than 75 percent, the odds of being college and career ready 
decrease by about one percent for each Achieve3000 activity completed. For students with an AFTS of 75 
percent or greater, the odds increase by four percent. 


Most schools do not have the resources to support 
an infrastructure conducive to effective teacher- 
delivered differentiated instruction. Even when 


Introduction 


Rooted in Vygotsky’s (1978) theory of Zone of 


Proximal Development (ZPD), defined as the 
difference between what a learner can do 
independently and what she or he can accomplish 
given guidance, differentiated instruction aims to 
tailor instruction to individual student needs. In 
contrast to traditional instruction, which targets 
middle, or average, achievers, differentiated 
instruction is a learner-centered model of instruction 
that recognizes the variance among classmates in 
their academic ability. 

Because of class size, however, successful 
teacher-delivered differentiated instruction is 
difficult to implement in schools (Lightweis, 2013). 
The ideal size for a group receiving teacher- 
delivered differentiated instruction is three to five, 
numbers that constitute a fraction of the typical 
American classroom size. 


support is available, teachers tasked with delivering 
differentiated instruction find the challenge of doing 
so greater than the potential returns to student 
achievement. In the short term, teacher-delivered 
differentiated instruction is labor-intensive, but its 
returns are cumulative and not immediately 
apparent, which, given all the responsibilities that 
the modern teacher has, often leads to a regression 
to familiar and often less effective teaching 
methods. 

Essential to the task of addressing diverse 
students’ educational needs is an acknowledgement 
of the limitations of the current education model, 
particularly its dependence on teachers as the sole 
responsibility-bearers for children’s educational 
experiences (Senge et al., 2000). Successfully 
shifting from teacher-centered to learner-centered 
education requires that students be treated as co- 


creators of the educational process. When learners 
actively participate in and are part of the decision- 
making process of their own education, teachers’ 
responsibilities primarily as knowledge providers is 
subsumed under the more effective role as 
facilitators of autonomous learning (Watson & 
Reigeluth, 2008). 

An indispensable tool for realizing the aim of 
learner-centered education and reducing disparities 
in educational opportunities, specifically as a 
consequence of differentiated instruction, is 
computer-, or web-based, technologies (Watson & 
Watson, 2011). As Jacob, Berger, Hart, and Loeb 


(2019) argue, growing Internet speed and 
accessibility to the world wide web can reduce many 
of the limitations that have historically 
disadvantaged poor _ students. Geographic 


limitations, or constraints, are removed because 
computer-based technologies scale easily so that 
they reach not just students in a single classroom or 
school, but across the district, the state, and the 
country. Correspondingly, in the same way that 
such technologies can benefit students, they can also 
assist in the professional development of less 
experienced teachers, allowing them access to more 
experienced teachers, from whom they can learn the 
knowledge and skills needed to be effective 
employing the varying kinds of instructional 
delivery, whether computer-based or face-to-face. 

The Houston Independent School District 
(HISD), to help its secondary students achieve 
lifelong success through literacy acceleration, has 
invested considerable monetary resources in 
Achieve3000®. A cloud-based solution accessible 
via computer or as an application on a tablet or 
smartphone, Achieve3000 provides daily 
differentiated instruction for nonfiction reading and 
writing adapted to students’ Lexile® reading level. 
When implemented with fidelity, whereby teachers 
encourage students to complete one to two or more 
reading or writing lessons per week (or 40 to 80 or 
more activities for the year) with an average first try 
score of 75 percent or greater, Achieve3000 has 
been shown to have a positive relationship with 
Lexile growth and college and career readiness 
(Achieve3000, 2019). Despite the district’s 
expenditure on Achieve3000 as a supplemental 
differentiated instruction-delivery tool, and 
notwithstanding the district’s evaluations of 
Achieve3000’s_ efficacy at improving student 
literacy and college and career readiness (Reeves, 
2014; Torres, 2015, 2016, 2018), usage, and 
specifically fidelity of use, among HISD students 
remains extremely low. 

The present study examines differences in both 
Lexile growth and college and career readiness by 
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fidelity of use among secondary students using 
Achieve3000 during the 2018-2019 academic year. 
Previous district reports on Achieve3000’s impact 
have attempted to include as many students in an 
analytic sample as possible, often focusing on 
achievement indicators besides the Achieve3000 
post-test Lexile or college and career readiness 
measure. This year’s study is restricted to those 
who, at the conclusion of the academic year, met 
most of the specific criteria outlined in the National 
Lexile Study, 2017-2018, including having 
completed both a pre-test Lexile assessment within 
the first 30 days of the start of the school year and a 
post-test Lexile assessment in the final 30 days of 
the school year (Achieve3000, 2019). 

Testing the relationship between fidelity of use 
and student Lexile growth and readiness for life 
after high school using only Achieve3000 data has 
been difficult because, historically, too few students 
have met both standards. The universe of students 
who now have access to Achieve3000, though, has 
expanded from only one secondary grade (grade 10) 
across 17 schools in 2013-2014 to all secondary 
grades (grades 6-12) across 100 schools in 2018- 
2019. Further, the number of students meeting both 
standards denoting fidelity of use, as well as meeting 
other criteria, has increased, making this analysis 
possible. 


Research Questions 


The aim, then, is to ascertain whether, net of 
student-level demographic factors, the combination 
of completed activities and average first-try score 
(AFTS) is: 


1. related to greater Lexile growth between 
the beginning and end of the academic 
year? 

2. related to greater odds of college and career 
readiness at the end of the academic year? 


Given the established relationship between 
Achieve3000 usage and student achievement shown 
in previous years’ research reports, it is 
hypothesized that there is a significant association 
between fidelity of use and both student Lexile 
growth and college and career readiness. 


Data and Method 


Sample 

Achieve3000 was made available to more than 
111,000 students enrolled in grades 6 through 12 
during the 2018-2019 academic year. The sample 


for this report, however, and consistent with the 
recent National Lexile Study, 2017-2018 
(Achieve3000, 2019), was limited to secondary 
students who (1) had a pre-test LevelSet Lexile 
measure within the first 30 days of the school year, 
(2) had an end-of-year post-test LevelSet Lexile 
measure within the final 30 days of school, and (3) 
completed at least one multiple choice activity. The 
first two restrictions ensure that students in the 
sample had at least 150 days between their pre-test 
and post-test measures, thus guaranteeing the 
maximum returns to those engaging the tool with 
fidelity. 

Students finishing the test too slowly, too quickly, 
or whose responses are patterned can yield Lexile 
scores that are not accurately aligned with their 
abilities, thus resulting in invalid scores. Students 
whose pre- or post-test LevelSet assessment was 
graded as invalid were therefore excluded from this 
study. Also excluded were students who received 
negative Lexile scores on either the pre-test or post- 
test assessment, an_ indication of poor 
implementation of the program. Finally, only those 
schools with at least 25 students were included in 
this analysis. The ultimate analytic sample, then, 
consisted of 4,934 students in 43 schools. Most 
students, or 3,141, were represented across 23 high 
schools, another 1,645 students were represented 
across 18 middle schools, and 148 students were 
represented in two combined-level schools. 


Measures 

Dependent Variables. To answer the research 
questions above, this study focused on two primary 
outcomes: growth and college and career readiness. 
Growth is a change score and is calculated as the 
difference between students’ post-test Lexile score 
earned at the end of the year and their pre-test Lexile 
score earned at the beginning of the year. 

College and career readiness is an indicator used 
to denote whether students are meeting the grade- 
specific Lexile level and are “On Track” to read at 
or above the college and career ready Lexile level of 
1300 by the time they graduate high school. 


Independent Variables. The main predictors 
for this study are the number of Achieve3000 
activities that students completed over the course of 
the academic school year and the average first-try 
score (AFTS) that students received on _ all 
Achieve3000 activities completed. The former 
predictor is a continuous variable ranging from one 
to 208 while the latter is a binary variable for which 
O denotes an AFTS less than 75 percent and 1 
denotes an AFTS of 75 percent or greater. 
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A primary moderating variable denoting students’ 
pre-test college and career readiness was included 
when assessing the effect of fidelity of use on post- 
test college and career. If a student had pre-test 
Lexile suggesting that he was college and career 
ready, he was coded 1 on the pre-test college and 
career readiness indicator; all others were coded 0. 

Additional student-level controls were entered for 
race/ethnicity, gender, economic disadvantage, 
special education status, English Learner (EL) 
status, at-risk status, overage status, and 
gifted/talented (G/T) status. A dummy variable was 
created for each category of each of these additional 
controls. 


Analytic Strategy 

To test the relationship between fidelity of use of 
Achieve3000 on student Lexile growth, and to 
answer the first research question, this study 
regressed the difference between end-of-year and 
beginning-of-year Lexile measure on completed 
activities, whether students had an AFTS of 75 
percent or greater, and all other covariates. Since 
growth, the continuous dependent variable in this 
study, is a change score denoting the difference 
between post- and pre-test Lexile scores, it was 
unnecessary to control for the pre-test Lexile scores 
when carrying out the linear regression analysis 
(Allison, 1990), as doing so tends to produce 
substantially inflated regression coefficients for 
other controls and can induce spurious statistical 
associations between those controls and the change 
in score (Glymour, Weuve, Berkman, Kawachi, & 
Robins, 2005). 

To understand more fully how fidelity of use of 
Achieve3000 usage is related to whether students 
are ready either for postsecondary education or the 
world of work, and to answer the second research 
question, this evaluation regressed students’ college 
and career readiness on completed activities, 
whether students had an AFTS of 75 percent or 
greater, and all other covariates. Because the 
outcome here was dichotomous, a logistic model 
was fit to ensure the output of a linear equation 
between 0 and 1. The result is a probability, or odds, 
of the occurrence of the event (college and career 
readiness = 1) based on the combination of known 
covariates. As the correlation between beginning- 
of-year and end-of-year college and career readiness 
was quite strong (r = .81, p < .001), each model 
adjusted, or controlled, for beginning-of-year 
college and career readiness to improve the 
efficiency of the analyses and to achieve robust, and 
more convincing, evidence of the relationship 
between the primary predictors and the outcome. 


Results 


Summary Statistics 

As stated above, Achieve3000 stresses the 
relevance of fidelity of use when assessing the tool’s 
impact on student growth and college and career 
readiness. Fidelity of use is defined as the minimum 
level of quantity and quality of use, or, as Table 1 
shows, the number of activities students completed 
over the course of the academic school year and the 
average first-try score (AFTS) students receive on 
those activities (Achieve3000, 2019). For this 
study, and consistent with the National Lexile Study, 
2017-2018 (Achieve3000, 2019), a binary indicator 
was created that codes the first two categories as 1 
(“Used with Fidelity’) and the bottom four 
categories as 0 (“Used without Fidelity”). 


Table 1. Fidelity of Use Descriptions 


Implementation 


Description 


Category 


High Quality, AFTS greater than or equal to 75% and 
Highest Quantity | on track to complete 80 lessons in a year 
(2+ activities/week) 

High Quality, AFTS greater than or equal to 75% and 
High Quantity on track to complete 40—79 lessons in a 


year (1+ activities/week) 


AFTS less than 75% and on track to 
complete 80 lessons in a year (2+ 
activities/week) 


Lower Quality, 
Highest Quantity 


Lower Quality, AFTS less than 75% and on track to 


High Quantity complete 40—79 lessons in a year (1+ 
activities/week) 
i High Quality, " AFTS greater than or equal to 75% and 


Lowest Quantity | on track to complete less than 40 lessons 


in a year (<1 activity/week) 


' AFTS less than 75% and on track to | 
complete less than 40 lessons in a year 
(<1 activity/week) 


Lower Quality, 
Lowest Quantity 


Source: Adapted from chart printed in the National Lexile Study, 2017- 
2018 (Achieve3000, 2019). 


Table A1 (p. 11) presents summary statistics by 
fidelity of use status. Focusing on the bottom 
portion of the table beneath the horizontal line 
dividing the table, it should be noted that most of the 
group percentages for race and ethnicity are similar 
between those who used Achieve3000 without 
fidelity and those who used it with fidelity. A 
slightly smaller percentage of whites and a slightly 
higher percentage of Hispanics used the program 
with fidelity than used it without fidelity; that is, 
these students completed at least 40 activities over 
the course of the academic year and had an AFTS of 
75 percent or greater. A slightly larger percentage 
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of economically disadvantaged students also used 
Achieve3000 with fidelity than used it without 
fidelity. There were much larger percentage 
differences in fidelity of use among the other 
demographic characteristics. Much — smaller 
percentages of special education, English learner, 
at-risk, and overage students used Achieve3000 
with fidelity than used it without fidelity, and a 
much greater percentage of gifted and talented 
students used Achieve3000 with fidelity than used it 
without fidelity. 

The top portion of Table A1 shows the means and 
standard deviations for number of activities 
completed, Lexile point growth, and percent college 
and career ready. Because the first of these is one 
of the two factors used to create the fidelity of use 
indicator, the wide difference in mean usage 
between the two categories for number of activities 
completed is expected. The important numbers to 
focus on are those for Lexile growth and college and 
career ready. Students who used Achieve3000 with 
fidelity had an average Lexile growth of about 169 
points compared to an average growth of only about 
51 points for those who used it without fidelity. 
About 68 percent of those using the program with 
fidelity had a gain of between 78 and 260 Lexile 
points. Sixty-eight percent of those using 
Achieve3000 without fidelity had a loss-gain of 
between -22 and 132 Lexile points. Figure 1 (p. 5) 
presents a visualization of the distribution of growth 
among the fidelity of use groups and for the total 
sample, with vertical lines drawn at the means. 

Figure Al (p. 12) shows the mean growth by 
fidelity of use status for middle and combined-level 
schools while Figure A2 (p. 13) shows the mean 
growth by fidelity of use status for high schools. In 
all but one school (Lamar High School) did students 
who used Achieve3000 with fidelity outperform 
their peers who used Achieve3000 without fidelity. 
Average school-level Lexile growth for students 
using the program with fidelity ranged from a low 
of 73 at Lamar High School to a high of 289 at 
Stevenson Middle School. Average school-level 
growth for students using the program without 
fidelity ranged from a low of 8 at the High School 
for Law and Justice to a high of 144 at Challenge 
Early College High School. 

Among middle and combined school students 
(Figure Al, p. 12), the largest Lexile point growth 
gap was witnessed at Stevenson, where students 
who used Achieve3000 with fidelity had mean 
growth of 289 points and those who used it without 
fidelity had a mean growth of 58, a 231-point gap. 
Kashmere High School students who used the 
program with fidelity had mean growth of 216 
Lexile points, while those who used the program 
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Figure 1. Kernel Density Plot of Achieve3000 Lexile Growth by Fidelity of Use 
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Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc 


Package in IBM Cognos. 


without fidelity had mean growth of 42, a 174-point 
gap (Figure A2, p. 13). That these two schools 
should also show large gains among students using 
Achieve3000 with fidelity is interesting. Relevant 
staff at both schools worked diligently to promote 
the use of Achieve3000 among its student body, 
Stevenson Middle School through a campus contest 
that ignited friendly competition, and Kashmere 
High School through the deployment of wraparound 
resources, with its holistic approach to providing 
support, to help that campus shed its Improvement 
Required Texas Education Agency (TEA) 
accountability rating. 

Returning to Table A1 (p. 11) and the comparison 
of means and standard deviations for college and 
career ready, only one in eight students who used 
Achieve3000 without fidelity met the grade-specific 
Lexile level standard denoting college and career 
ready; one in two students who used the program 
with fidelity did so. Figure 2 (p. 6) presents a bar 
graph to help visualize these differences. As is 
readily apparent, completing 40 activities or more in 
tandem with achieving an AFTS of 75 percent or 


greater is associated with being ready for college, 
the workforce, or the military at the conclusion of 
the academic year. 

While these descriptive statistics and figures 
presented thus far are instructive, they do not control 
for moderating factors that may change the 
relationship between fidelity of use and either Lexile 
growth or college and career readiness. For that, 
multivariate regression models are used. 


Fidelity of Use and Lexile Growth 

Table A2 (p. 14) shows the results of the 
multivariate linear mixed regression parameters for 
two-level models examining the impact of 
Achieve3000 fidelity of use (ie, number of 
activities completed, AFTS of 75% or greater, and 
the interaction between the two) on student Lexile 
growth for all secondary-grade students. Model 1 
estimated the unconditional model to highlight the 
interclass correlation coefficient (ICC) for Lexile 
growth, or the ratio of the between-school variance 
to the total variance. Since the between-school 
variance is 738.045, and the within-school variance 
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Figure 2. Percentage of Students Who Were Either College and Career Ready or Not College and 
Career Ready by Fidelity of Use of Achieve3000 
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is 6619.259, the ICC for Lexile growth is 
738.045/(738.045 + 6619.259), or 0.100. This 
means that exactly 10 percent of the variance in 
Lexile growth is due to school clusters and that the 
correlation among students in the same school for 
Lexile growth is 0.100. 

Model 2 examined the relationship between the 
two primary predictors, activities completed and 
AFTS of 75 percent or greater, and Achieve3000 
Lexile growth. The between-school variance 
decreased by 65 percent between Models 1 and 2, 
from 738.045 to 253.740, while the within-school 
variance decreased 23.6 percent, from 6619.259 to 
5056.993. These decreases denote the amount of 
variance explained due to the inclusion of the 
primary fidelity of use predictors. The correlation 
among students in the same school for Lexile 
growth is 253.740/(253.740 + 5056.993), or 0.048, 
suggesting that about 5 percent of the variance in 
Lexile growth is due to school clusters. 

Model 3 added an interaction between activities 
completed and AFTS of 75 percent or greater. This 
suggests that approximately 73.5 percent (i.e., the 
variance reduction from 738.045 to 195.679) of the 


between-school variance in Achieve3000 Lexile 
growth is explained by activities completed, AFTS, 
and the interaction between the two. About 27.4 
percent (i.e., the variance reduction from 6619.259 
to 4803.023) of within-school variance in Lexile 
growth is explained by the same. 

Model 4 investigated the effect of Achieve3000 
fidelity of use on Lexile growth after controlling for 
student characteristics. The reduction in the 
variance components from Model 1 to Model 4 
shows that the included covariates accounted for 
76.0 percent (i.e., variance reduction from 738.045 
to 177.081) of the between-school variance and 28.4 
percent (i.e., the variance reduction from 6619.259 
to 4738.746) of the within-school variance. Mean 
Lexile growth was 21.03, net of the effects of any 
controls. All other things being equal, Lexile 
growth did not vary significantly by race/ethnicity, 
gender, economically disadvantaged status, special 
education status, at-risk status, overage status, or 
gifted and talented status. Interestingly, English 
learners experienced mean Lexile growth of 15.72 
points more than their non-English learner peers. 
The combination of activities completed, AFTS, and 


the interaction between the two revealed that, for 
two students with an AFTS of less than 75 percent, 
the one completing 40 activities for the year would 
gain about 14 Lexile points more than the one who 
completed no activities. Likewise, for two students 
with an AFTS of 75 percent or greater, the one 
completing 40 activities over the year would gain 
about 77 more Lexile points than the one who 
completed no activities. These values are in 
addition to the mean of 21.03 for students with an 
AFTS less than 75 percent and in addition to the 
mean of 21.03 + 40.398 for students with an AFTS 
of 75 percent or greater. Figure 3 illustrates the 
interaction term shown in Model 4. 


Fidelity of Use and College and Career Readiness 

Table A3 (p. 15) fit four multilevel mixed effects 
regression models showing the relationship between 
fidelity of use and odds of college and career 
readiness for all secondary-grade students. Model 1 
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presents the unconditional, or null, model, which 
incorporates only school-specific random effects to 
model the between-school variation in college and 
career readiness at the conclusion of the academic 
year. Model 2 added the primary fidelity of use 
characteristics, activities completed and AFTS, as 
well as a control for college and career readiness at 
the beginning of the year. Model 3 added the 
interaction between activities completed and AFTS. 
The final model (Model 4) incorporated student- 
level characteristics. Because they are easier to 
interpret, odds ratios are presented instead of the 
default estimated regression coefficients, often 
shown as log odds. 

The unconditional model revealed an intercept of 
0.103 and an estimated variance on the random 
effects of 7.741. Turning to Model 2, the odds ratio 
for beginning-of-year college and career readiness 
suggests that, when comparing two students who 
have different values for readiness, but who are 


Figure 3. Marginal Linear Prediction of the Interaction between Number of Activities Completed 


and AFTS on Achieve3000 Lexile Growth 
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Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc 
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similar in every other respect, the odds of being 
college and career ready at the end of the year is 405 
times higher for those who were college and career 
ready at the beginning of the year than for those who 
were not ready. The odds of being college and 
career ready at the end of the year is 27 times higher 
among students who had an AFTS of 75 percent or 
greater relative to those who had an AFTS of less 
than 75 percent. Also, for each Achieve3000 
activity completed, the odds of college and career 
readiness increase by 2 percent, double the impact 
shown in the National Lexile Study, 2017-2018 
(Achieve3000, 2019). 

Model 3, which added the interaction between 
activities completed and AFTS, showed that starting 
the year college and career ready was a strong 
predictor for ending the year college and career 
ready. Such students were 379 times more likely to 
be ready for college, the workplace, or the military 
than were their peers who were not college and 
career ready at the beginning of the year. Taking the 
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coefficient for the interaction between activities 
completed and AFTS, the results revealed that for 
each Achieve3000 activity completed among 
students with an AFTS less than 75 percent remain 
flat. For students with an AFTS of 75 percent or 
greater, the impact of each activity completed 
increases the odds of being college and career ready 
by about 4 percent. 

Controlling for student characteristics only 
slightly augments the findings from Model 3. 
Focusing on the interaction term under Model 4, for 
students with an AFTS less than 75 percent, the odds 
of being college and career ready decrease by about 
one percent for each Achieve3000 activity 
completed. For students with an AFTS of 75 
percent or greater, the odds increase by four percent. 
Being college and career ready at the start of the 
school year is still highly predictive of being college 
and career ready at the end of the school year. 
Figure 4 presents, by college and career readiness 
status at the beginning of the year, a graphical 


Figure 4. Marginal Predicted Mean of the Interaction between Number of Activities and AFTS on 
End-of-Year College and Career Readiness by Beginning-of-Year College and Career 


Readiness 
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Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc 


Package in IBM Cognos. 


representation of the interaction term in Model 4. 
Note that students who were not college and career 
ready at the beginning of the year, but who used 
Achieve3000 with fidelity (i-e., completed at least 
AO activities and had an AFTS of 75 percent or 
greater), improved their odds of being college and 
career ready by the end of the year (green dotted 
line). While being on track for college and career 
readiness at the beginning of the year meant higher 
odds of being ready at the end of the year, if it was 
accompanied by low quality usage of Achieve3000, 
the odds could decline (yellow dashed line). 


Conclusion 


This study set out to answer the questions of 
whether, among HISD secondary students, fidelity 
of use of a computer-based differentiated literacy 
instruction program was associated with greater 
Lexile growth and higher odds of college and career 
readiness. The results affirm that it does. When 
students work diligently toward completing at least 
40 Achieve3000 activities throughout the academic 
year (defined as high quantity), and when they do so 
with an average first-try score (AFTS) of 75 percent 
or greater (defined as high quality), they achieve 
greater growth and are more likely to be college 
ready, net of moderating factors. 

District curriculum professionals, — school 
administrators, and teachers, of course, play an 
important role in making sure students know about 
and are using the technology-based tools at their 
disposal. When the tools assist teachers in 
delivering the type of instruction designed to meet 
the needs of diverse learners, as Achieve3000’s 
computer-based differentiated instruction does, it 
should be a goal to promote its use by students, as 


doing so _ facilitates effective differentiated 
instruction themselves. Unfortunately, despite 
HISD’s expenditure on Achieve3000, usage 


remains low among the approximately 111,000 
secondary students to whom it is available. As such, 
Achieve3000’s benefits are not widely experienced. 


Recommendations 

To improve usage of Achieve3000 among HISD’s 
secondary students, two courses of action are 
recommended. First, curriculum professionals, in 
concert with senior school personnel and teachers, 
need to ensure that students are completing the pre- 
test within the first 30 days of the beginning of the 
school year and the post-test in the last 30 days of 
the school year. Second, students must be regularly 
encouraged to complete one to two, preferably 
more, Achieve3000 activities per week throughout 
the year. The first course of action requires planning 


Achieve3000 Fidelity of Use 


at the classroom level, but teachers need to know 
that Achieve3000 is effective and why (i.e., non- 
fiction builds literacy and knowledge across content 
areas). Its use among students will augment their 
goal of fruitful instruction aimed at all students, 
including those reading below their grade level. 

The second course of action builds on the first. 
Once teachers have buy-in, they will strive to 
remind students to complete Achieve3000 activities, 
fully cognizant that doing so will absolutely yield 
positive results for their students. 
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Table Al. Summary Statistics by Fidelity of Use 


Achieve3000 Fidelity of Use 


Number of Activities Completed 
Lexile Growth (in points) 


College and Career Ready (%) 


Race/Ethnicity 
White (%) 
Black (%) 
Hispanic (%) 
Asian/Pacific Islander (%) 
Other Race (%) 
Gender 
Female (%) 
Male (%) 
Economically Disadvantaged (%) 
Special Education (%) 
English Learner (%) 
At-Risk (%) 
Overage (%) 


Gifted/Talented (%) 


Appendix 
Used Without Fidelity | Used With Fidelity Total 
(n = 4,532) (n = 402) (n = 4,934) 

M SD M SD M SD 
28.3 20.0 57.8 16.8 30.7 21.4 
50.9 77.0 168.6 90.6 60.5 84.5 
12.3 32.9 51.7 50.0 15.5 36.2 
2.9 16.9 2.0 14.0 2.9 16.7 
21.4 41.0 19.9 40.0 21.3 40.9 
73.5 44.2 76.1 42.7 73.7 44.0 
1.6 12.5 1.7 13.1 1.6 12.6 
0.6 7.6 0.2 5.0 0.5 7.4 
49.1 50.0 60.4 49.0 50.0 50.0 
50.9 50.0 39.6 49.0 50.0 50.0 
88.7 31.6 93.0 25.5 89.1 31.2 
9.0 28.7 2.2 14.8 8.5 27.9 
27.0 44.4 9.7 29.6 25.6 43.6 
48.9 50.0 23.4 42.4 46.8 49.9 
25.2 43.4 13.7 34.4 24.2 42.9 
11.7 32.2 22.9 42.1 12.6 33.2 


Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc Package 


in IBM Cognos. 


Note: Fidelity of Use refers to whether a student had both an average first try score of 75 or greater (high quality) AND 
40 or more (high quantity) completed Achieve3000 activities (coded 1) during the academic year. All other students, 
who had either a combination of low quantity and high quality, or low quality and high quantity, were coded 0. 
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Figure A1. Mean Achieve3000 Lexile Growth by Fidelity of Use and Middle and Combined School 
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Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc 
Package in IBM Cognos. 

Note: An asterisk denotes a combined-level school that educates all secondary students (i.e., those in grades 6 
through 12). 
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Figure A2. Mean Achieve3000 Lexile Growth by Fidelity of Use and High School 
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Source: Achieve3000 Student Data 2018-2019 and Student Demographic Data 2018-2019 from SIS Ad Hoc 
Package in IBM Cognos. 
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Table A2. Estimated Effects of Completed Achieve3000 Activities and Average First-Try Score on 
Lexile Growth for All Secondary Grades 


Model 1 Model 2 Model 3 Model 4 
Lexile Growth 60.374°"" 13.872" 28.409""" 21.030" 
(4.410) (3.309) (3.174) (7.187) 
Activities Completed 0.677" 0.203" 0.171" 
(0.057) (0.063) (0.063) 
AFTS of 75% or Greater 89.047""" 37.751" 40.398""" 
(2.334) (3.866) (3.905) 
Activities Completed x AFTS of 75% or 1.709"™" 1.741" 
Greater (0.104) (0.104) 
Race/Ethnicity (Ref. = White) 
Black -2.115 
(6.402) 
Hispanic -4.452 
(6.067) 
Asian/Pacific Islander 5.325 
(9.795) 
Other Race 7.715 
(14.559) 
Male (Ref. = Female) -0.478 
(1.986) 
Economically Disadvantaged 4.716 
(3.323) 
Special Education -6.376 
(3.708) 
English Learner 15.720°" 
(3.128) 
At-Risk 4.052 
(2.782) 
Overage 3.842 
(2.553) 
Gifted/Talented 1.557 
(3.271) 
Variance Components 
Between-School Variance 738.045" = 253.740" = 195.679" —-177.081""" 
(90.574) (35.653) (28.332) (26.154) 
Within-School Variance 6619.259"™" 5056.993"™" 4803.023°™" 4738.746"" 
(66.929) (51.138) (48.563) (47.913) 
AIC 57517.4 56164.4 55905.4 55858.2 
BIC 57536.9 56197.0 55944.4 55968.7 
Observations 4934 4934 4934 4934 


Standard errors in parentheses 
*p <0.05, ™ p < 0.01,” p < 0.001 
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Table A3. Estimated Effects of Completed Achieve3000 Activities and Average First-Try Score on 
College and Career Readiness for All Secondary Grades (Presented as Odds Ratios) 


Model 1 Model 2 Model 3 Model 4 


College and Career Readiness (End-of-Year) 0.103" 0.003" 0.0077" 0.024""" 
(0.024) (0.001) (0.002) (0.013) 
College and Career Readiness (Beginning-of-Y ear) 405.308" 378.705"™" = 283.852" 
(94.382) (84.355) (63.546) 
Activities Completed 1.023" 0.991 0.990 
(0.003) (0.007) (0.007) 
AFTS of 75% or Greater 27.138" —5.846""* 4.315°™ 
(5.863) (1.834) (1.385) 
Activities Completed x AFTS of 75% or Greater 1.047" 1.051°™ 


(0.008) (0.009) 
Race/Ethnicity (Ref. = White) 


Black 0.543 
(0.249) 

Hispanic 0.532 
(0.233) 

Asian/Pacific Islander 1.451 
(1.049) 

Other Race 1.401 
(1.617) 

Male (Ref. = Female) 1.071 
(0.169) 

Economically Disadvantaged 0.742 
(0.171) 

Special Education 0.702 
(0.363) 
English Learner 0.444" 
(0.160) 

At-Risk 0.679 
(0.151) 

Overage 0.606 
(0.159) 
Gifted/Talented 1.776" 
(0.345) 

Variance Component 

Between-School Variance 7.741" 1.340" 1.270 1.126 
(4.318) (0.184) (0.161) (0.098) 
AIC 3715.2 1337.1 1299.9 1272.5 
BIC 3728.2 1369.6 1338.9 1383.1 

Observations 4934 4934 4934 4934 


Exponentiated coefficients; Standard errors in parentheses 
“p< 0.05, ™ p < 0.01,” p < 0.001 
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